﻿<!DOCTYPE HTML>
<!--------------------------------------------------------------------------->  
<!--                           INTRODUCTION                                

 The CodeProject article submission template (HTML version)

Using this template will help us post your article sooner. To use, just 
follow the 3 easy steps below:
 
     1. Fill in the article description details
     2. Add links to your images and downloads
     3. Include the main article text

That's all there is to it! All formatting will be done by our submission
scripts and style sheets. 

-->  
<!--------------------------------------------------------------------------->  
<!--                        IGNORE THIS SECTION                            -->
<html>
<head>
<title>CodeProject</title>
<Style>
BODY, P, TD { font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt }
H2,H3,H4,H5 { color: #ff9900; font-weight: bold; }
H2 { font-size: 13pt; }
H3 { font-size: 12pt; }
H4 { font-size: 10pt; color: black; }
PRE { BACKGROUND-COLOR: #FBEDBB; FONT-FAMILY: "Courier New", Courier, mono; WHITE-SPACE: pre; }
CODE { COLOR: #990000; FONT-FAMILY: "Courier New", Courier, mono; }
</style>

<link type="text/css" rel="stylesheet" 
      href="https://codeproject.global.ssl.fastly.net/App_Themes/CodeProject/Css/Main.min.css">
</head>
<body bgcolor="#FFFFFF" color=#000000>
    <!--------------------------------------------------------------------------->
    <!-------------------------------     STEP 1      --------------------------->
    <!--  Fill in the details (CodeProject will reformat this section for you) -->

<!--<pre>
Title:       Enter Article Title Here
Author:      Enter your Code Project User Name (or suggest one if you're not a member)
Email:       Enter your Code Project E-mail Login (or the email you wish to use if not a member)
Language:    Enter the Languages that Apply to Your Article (C# 3.0, etc.)
Platform:    Enter the Platforms that Apply to Your Article (Windows, etc.)
Technology:  Enter the Technologies that Apply to Your Article (WPF, etc.)
Level:       Pick ONE: Beginner/Intermediate/Advanced
Description: Enter a brief description of your article
Section      Enter the Code Project Section you Wish the Article to Appear
SubSection   Enter the Code Project SubSection you Wish the Article to Appear
License:     Enter the license (<a href="http://www.codeproject.com/info/licenses.aspx">CPOL, CPL, MIT, etc</a>)
</pre>-->

<style>
table.mnist  {
  border: 1px solid blue;
  border-collapse: collapse;
}
table.mnist th
{
  border: 2px solid blue;
  border-collapse: collapse;

}
table.mnist tr
{
  border: 1px solid blue;
  border-collapse: collapse;
}
table.mnist td
{
  border: 1px solid blue;
  border-collapse: collapse;
  padding-left:5px;
  padding-right:5px
}
table.mnist td.header
{
  font-weight:bold;
}
table.mnist thead td
{
    font-weight:bold;
}
</style>

    <!-------------------------------     STEP 2      --------------------------->
    <!--  Include download and sample image information.                       -->

    <ul class=download>
        <li><a href="https://github.com/sdg002/MNISTpng">Download source code from Github</a></li>
    </ul>

    <p>
        <img src="images/Home_using_ppt.PNG" 
             style="width:400px; height:200px">
    </p>

    <!-------------------------------     STEP 3      --------------------------->
    <!--  Add the article text. Please use simple formatting (<h2>, <p> etc)   -->

    <h2 id="toc">Table of Contents</h2>
    <ul>
        <li>
            <a href="#intro">Introduction</a>
        </li>
        <li>
            <a href="#tensorflow">About Tensorflow</a>
        </li>
        <li>
            <a href="#back">Background</a>
        </li>
        <li>
            <a href="#mnist">What is MNIST? Why MNIST</a>
        </li>
        <li>
            <a href="#deeplearning">Deep learning</a>
            <ul>
                <li><a href="#deeplearning_perceptrons">Perceptrons</a></li>
                <li><a href="#deeplearning_singleperceptrons">Single perceptron</a></li>
                <li><a href="#deeplearning_multi_perceptrons">Multi layer perceptrons</a></li>
                <li><a href="#deeplearning_cnn">Convolutional Neural Networks</a></li>

            </ul>
        </li>
        <li>
            <a href="#tensorflowsharp">About TensorflowSharp-Using Tensorflow from a C# application</a>
        </li>
        <li>
            <a href="#tensorflowgpu">Using Tensorflow with GPU</a>
            <ul>
                <li><a href="#tensorflowgpu_overview">Overview</a></li>
                <li><a href="#tensorflowgpu_whatworkedforme">GPU/NVDIA-What worked for me?</a></li>
            </ul>
        </li>
        <li>
            
            <a href="#tensorflowgpu">Training a CNN model using TensorFlow and Python</a>            
            <ul>
                <li><a href="#trainingusingpython_cnnarchitecture">CNN Archictecture</a></li>
                <li><a href="#trainingusingpython_loadimages">Image files used for training</a></li>
                <li><a href="#trainingusingpython_loadimagespython1">1-Python script (MnistImageLoader.py)</a></li>
                <li><a href="#trainingusingpython_loadimagespython2">2-Loading the training images (TrainMnistFromFolder.py)</a></li>
                
                <li><a href="#trainingusingpython_createmodel">3-Create the CNN model(TrainMnistFromFolder.py)</a></li>
                <li><a href="#trainingusingpython_trainmodel">4-Train the model(TrainMnistFromFolder.py)</a></li>
                <li><a href="#trainingusingpython_savemodel">5-Save the model to file(TrainMnistFromFolder.py)</a></li>
                <li><a href="#trainingusingpython_results">6-Results</a></li>
                
            </ul>
        </li>
        <li>
            <a href="#csharpconsole">Using Tensorflow in a C# console application</a>
            <ul>
                <li><a href="#csharpconsole_overview">Overview</a></li>
                <li><a href="#csharpconsole_create">1-Create a console application</a></li>
                <li><a href="#csharpconsole_loadmodel">2-Load the trained model file</a></li>

                <li><a href="#csharpconsole_imagetensor">3-Utils.ImageToTensorGrayScale</a></li>
                <li><a href="#csharpconsole_quantized">4-Utis.Quantized</a></li>

                <li><a href="#csharpconsole_results">5-Results</a></li>
                
            </ul>
        </li>
        <li>
            <a href="#usingthecode">Using the code</a>
            <ul>
                <li><a href="#usingthecode_github">Github</a></li>
                <li><a href="#usingthecode_structure">1-Solution structure</a></li>
                <li><a href="#usingthecode_pythontrainer">2-Python Trainer</a></li>
                <li><a href="#usingthecode_consoleapptester">3-ConsoleAppTester</a></li>
            </ul>
        </li>
        <li>
            <a href="#pointsofinterest">Points of interest</a>
        </li>
    </ul>

    <h2><a id="intro"></a>Introduction</h2>
    <p>In the field of pattern recognition, deep neural networks have gained prominence in the last 5 years. This can be largely attributed to availability of 
        cheaper hardware, programming libraries and labelled data. Deep neural networks or Convolutional neural networks (CNN) if trained properly 
        can give spectacular results. TensorFlow from Google is one the very popular libraries that implement some of these complicated algorithms</p>
    <p>
    In this article I am going to demonstrate how to train a CNN model to recognize handwritten digits from the MNIST database. This will be followed by a 
    C# console application which will consume the trained model to actually classiy the test images from the MNIST dataset. The objective of this article
        is to demonstrate how to make the best use of Python for training a model and .NET to build a hypothetical end user application which consumes the trained model.
    </p>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h2><a id="tensorflow"></a> About TensorFlow</h2>
    <h3>Tensorflow native library</h3>    
    <pre>
        ///
        ///https://www.tensorflow.org/install/lang_c
        ///The windows native implementation is downloadable as a single ZIP and structured as follows
        ///

        include
        --------
                |
                |
                |
                --c_api.h
                |
                |
        lib
        --------
                |
                |
                --tensorflow.dll
                |
                |
                --tensorflow.lib


    </pre>

    <h3>Tensorflow bindings for Python and C#</h3>
    <p>Tensorflow is implemented as C/C++ dynamic link library. Platform specific binaries are available in a ZIP file. Bindings in various languages are provided on top of this library. These are language specific wrappers which invoke the native libraries.
    Python is perhaps one of the most versatile programming layers built on top of the native TensorFlow implementations. TensorFlowSharp is the .NET wrapper over TensorFlow.
    </p>
    <pre>
                        TensorFlow(C/C++)
                        ----------------
                            |
                            |
        ------------------------------------------------
        |                                               |
        |                                               |
        |                                               |
      Python                                        TensorFlowSharp(C#)
      ------                                        -------------------
  (train model)                             (use model in client application)
    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h2 id="back">Background</h2>
    <ul>
        <li><strong>Python</strong> - I have used Python for training a CNN model using the MNIST dataset of handwritten digits. A basic knowlege of Python would be essential.
            I have used <strong>Visual Studio Code (1.36.1)</strong> for the Python scripts. You can use any Python editor that suits you.
        </li>
        <li>
            I have used <strong>Visual Studio 2017</strong> for the simple Console application which consumes the trained model and classifies the test images
        </li>
        <li>
            In this article I have used the <strong>GPU</strong> version of the Tensorflow for to improve the speed of learning. You would need  a <strong>GPU</strong> enabled desktop.
        The reader is cautioned that coding with the GPU requires additional <strong>CUDA</strong> libraries and drivers to be installed. This article also assumes that the reader
        is familiar with the basic principles of <strong>Deep Convolution Neural Networks</strong>.
        </li>
    </ul>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    

    <h2><a id="mnist"></a> What is MNIST? Why MNIST?</h2>
    <h3>Overview</h3>
    <p>MNIST database is a collection of handwritten digits (0-9). This comprises of 60,000 training and 10,000 testing images. Each of the images 
        is 28 pixels wide and 28 pixels high and all the images are in gray scale. 
        In the world of machine learning and computer vision, MNIST has become the de facto
        standard to test any new paradigm. (Reference <a href="http://yann.lecun.com/exdb/mnist/">http://yann.lecun.com/exdb/mnist/</a>)
        <img src="images/3_expanded.PNG" style="width:50%;height:50%;"/>
    </p>
    <h3>Example pictures</h3>
    <p>
        <img src="images/0.png"/>
        <img src="images/1.png"/>
        <img src="images/2.png"/>
        <img src="images/3.png"/>
        <img src="images/4.png"/>
        <img src="images/5.png"/>
        <img src="images/6.png"/>
        <img src="images/7.png"/>
        <img src="images/8.png"/>
        <img src="images/9.png"/>
    </p>
    <h3>Distribution of pictures</h3>
    <table class="mnist">
        <thead>
            <tr>
                <td></td>
                <td>0</td>
                <td>1</td>
                <td>2</td>
                <td>3</td>
                <td>4</td>
                <td>5</td>
                <td>6</td>
                <td>7</td>
                <td>8</td>
                <td>9</td>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td class="header">Training</td>
                <td>5923</td>
                <td>6742</td>
                <td>5985</td>
                <td>6131</td>
                <td>5842</td>
                <td>5421</td>
                <td>5918</td>
                <td>6265</td>
                <td>5851</td>
                <td>5949</td>
            </tr>
            <tr>
                <td class="header">Testing</td>
                <td>980</td>
                <td>1135</td>
                <td>1032</td>
                <td>1010</td>
                <td>982</td>
                <td>892</td>
                <td>958</td>
                <td>1028</td>
                <td>974</td>
                <td>1009</td>

            </tr>
        </tbody>
    </table>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h2><a id="deeplearning"></a> Deep learning</h2>
    <h3 id="deeplearning_perceptrons">Perceptrons</h3>
    <p>
    In the 1940s and 50s the idea of a very basic mathematical neuron began to take shape. Researchers (McCulloch, Pitts and Rosenblatt) drew inspiration from the working of a biological neuron. Neurons are the building blocks of the nervous system. An average human brain has several billions of neurons indirectly connected to each other through synapses.
    They envisioned an individual neuron to behave like a straight-line classifier. The electric signals flowing in through the 
    dendrites represented a real-life signal (vector) and the output signal would represent a binary (on/off) state of classification. Frank Rosenblatt (1962) took the design of McCulloch and Pitts neuron a step forward by proposing the design of the linear perceptron in his book Principles of Neurodynamics (published 1962, Section “Completely linear perceptrons”). 
    </p>
    <img src="images/BioNeuron.PNG"/><br />
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="deeplearning_singleperceptrons">Single perceptron</h3>
    <img src="images/NeuronSingleNode.PNG"/>
    <p>The blue circle represents the equation of a straight line in the form of a.x+b.y+c=0. </p>
    <img src="images/Neuron_LinearClassifier.PNG" />
    <p>Given two classes of points X and O, which are linearly separable, you can find a straight line which divides the 2 classes. 
        If you feed in the coordinates of the points in class X to the equation a.x+b.y+c and then do the same for all points in class O
        then you are going to see that all points in class X produce a positive value , while all points in class O produce a negative value (or vice versa). 
        The change of sign could be other way round , depending on the constants a,b and c. Nevertheless, this is the overarching principle 
        which goes makes the Perceptron behave as a linear classifier
    </p>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="deeplearning_multi_perceptrons">Multi layer perceptrons</h3>
    <img src="images/Neuron_XOR.PNG"/>
    <p>
    If we are unable to find a single line that separates the classes X and O , as in the case of the famous XOR problem, then we could 
    cascade multiple linear classifiers.
    </p>
    <img src="images/XOR_TruthTable.PNG"/>
    <br />
    <img src="images/MultiLayerPerceptron.PNG"/><br/>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="deeplearning_cnn">Convolutional Neural Networks</h3>
    <p>
        Deep learning takes a multi layer perceptron a step forward by combining feature extraction and hyperplane discovery.
        The features are extracted by layers of filters. For a thorough treatise on this subject, the reader is requested to follow 
        Andrew Ng's tutorials.
    </p>
    <iframe width="560" height="315" src="https://www.youtube.com/embed/ArPaAX_PhIs" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
    <br/>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    

    


    <h2><a id="tensorflowsharp"></a>TensorflowSharp - Using Tensorflow from a C# application</h2>
    <p>TensorFlowSharp is a .NET wrapper over the unmanaged native libraries of TensorFlow. This is the outcome of the pioneering work 
    done by <a href="https://github.com/migueldeicaza/TensorFlowSharp"><strong>Miguel de lcaza</strong></a>. 
        TensorFlowSharp can consume a CNN model that was trained using Python and this opens up the possibility to create exciting end user applications.
    
    </p>
    <pre>
        nuget install TensorFlowSharp
    </pre>
    <pre>
        ///
        ///Skeletal code using TensorFlowSharp
        ///
        byte[] buffer = System.IO.File.ReadAllBytes(modelfile);
        using (var graph = new TensorFlow.TFGraph())
        {
            graph.Import(buffer);
            using (var session = new TensorFlow.TFSession(graph))
            {   
                /*
                1)pick a test image
                2)Created tensor object using this image
                */
                var runner = session.GetRunner();
                runner.AddInput(...,tensor,)
                runner.Fetch(...)
                var output = runner.Run();
            }
        }
    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>


    <h2 id="tensorflowgpu">Using Tensorflow with GPU</h2>
    <h3 id="tensorflowgpu_overview">Overview</h3>
    <pre>
                                Python script
                                --------------
                                        |
                                        |
                                        |

                              TensorFlow GPU package
                              ----------------------
                                        |
                                        |
                                        |
                                      cuDNN
                                      -----
                                        |
                                        |
                                        |
                                  CUDA Toolkit
                                  --------------
                                        |
                                        |
                                        |
                                     Drivers
                                     -------
                                        |
                                        |
                                        |
                                       GPU
                                       ---
                                
    </pre>
    <h3 id="tensorflowgpu_whatworkedforme">GPU/NVDIA-What worked for me?</h3>
    <p>
        When using TensorFlow for training, you have the choice of using either the CPU package or the GPU package. The GPU is preferred because the training speed is significantly faster.  
        You will need the correct version of NVIDIA drivers and CUDA libraries. As a rule of thumb,the version of NVIDIA drivers should match the current version of TensorFlow. At the time of writing this article, I have used the python package TensorFlow-GPU 1.14.0.
        I would caution the reader that my experience with installing the drivers and getting TensorFlow GPU to work was less than smooth.
    </p>
    <ul>
        <li>
            Update the version of the NVIDIA drivers. I did not install the through NVIDIA web site. I updated the display adapter through the Windows Device Manager user interface.
            The version 24.21.14.1131  worked for me.

            <img src="images/DeviceManagerDisplayAdapter.png" style="border:1px solid gray"/>
        </li>
        <li>
            Install CUDA Toolkit 10.0 version
        <li>
            Install cuDNN SDK version 7.6.2 . I chose the Windows 10 edition. I copied over cudnn64_7.dll to %ProgramFiles%\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin
        </li>
        <li>
            Python package Tensorflow 1.14
        </li>
        <li>
            Python package Keras 2.2.4
        </li>
        <li>
            Python package Numpy 1.16.1
        </li>
    </ul>
    <a href="#toc" title="Back to Table of Contents">Top</a>

    <h2><a id="trainingusingpython"></a> Training a CNN model using TensorFlow and Python</h2>
    <h3 id="trainingusingpython_cnnarchitecture">CNN architecture</h3>
    <pre>
                                
                                 Input layer (28X28,1 channel)
                                 -----------------------------
                                            |
                                            |
                                            |           
                                 Convolution layer (5X5,20,RELU)
                                --------------------------------
                                            |
                                            |
                                            |           
                                Max Pool layer (2X2,stride=2)
                               ------------------------------
                                            |                              
                                            |                              
                                            |     
                                Convolution layer (5X5,50,RELU)
                               --------------------------------
                                            |                              
                                            |                              
                                            |                              
                                Max Pool layer (2X2,stride=2)
                                -----------------------------
                                            |                              
                                            |                              
                                            |     
                                         Flatten
                                        ---------
                                            |                              
                                            |                              
                                            |     
                                Dense layer (500 nodes,RELU)
                                ----------------------------
                                            |                              
                                            |                              
                                            |     
                                Dense layer (10 nodes,RELU)
                                ----------------------------
                                            |                              
                                            |                              
                                            |     
                                Output layer(Softmax)
                                ----------------------
    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="trainingusingpython_loadimages">Image files used for training</h3>
    <p>
        MNIST dataset can be readily accessed from the <a href="https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html">scikit-learn</a> package.
        However in this tutorial I have demonstrated how to load the images from disk. The individual PNG files are made available in the accompanying project MNISpng.csproj.        
        The python script MnistImageLoader.py will enumerated over the the directory structured and build a list of training/testing images. The parent folder of each PNG file will provide the 
        training label (0-9). 
    </p>
    <pre>
        MNIST
        -----
                |
                |
                training.zip
                -----------
                |    |
                |    |
                |    |--(folders 0 to 9)
                |           |
                |           |
                |           |_0
                |           |
                |           |
                |           |_1
                |           |
                |           |
                |           |_2
                |           .
                |           .
                |           ._9
                |
                |
                testing.zip
                -----------
                     |
                     |
                     |--(folders 0 to 9)
    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="trainingusingpython_loadimagespython1">1-Python script (MnistImageLoader.py)</h3>
    <pre>
        #
        #Load images and labels. Returns a tuple of image data,label
        #
        def load_images(path_in):
                filenames = glob.glob(path_in)
                images=[] 
                labels=[] #labels for each training file
                filenames = glob.glob(path_in)
                for filename in filenames:
                        #get the parent folder from the full path of the file /mnist/blah/training/3/34348.png
                        fulldir=os.path.dirname(filename)
                        parentfolder=os.path.basename(fulldir)
                        imagelabel=int(parentfolder)
                        labels.append(imagelabel)
                        img = get_im(filename)
                        images.append(img)
                return images,labels

        #
        #The output from load_images() is further refined
        #
        def ReShapeData(data,target,numclasses):
                data_out = np.array(data, dtype=np.uint8)
                target_out = np.array(target, dtype=np.uint8)
                data_out = data_out.reshape(data_out.shape[0],  28,28)
                data_out = data_out[:, :, :, np.newaxis]
                data_out = data_out.astype('float32')
                data_out /= 255
                target_out = np_utils.to_categorical(target_out, numclasses)
                return data_out,target_out

    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="trainingusingpython_loadimagespython2">2-Loading the training images (TrainMnistFromFolder.py)</h3>
    <p>The master python script TrainMnistFromFolder.py will call the functions <strong>load_images</strong> and <strong>ReShapeData</strong>.</p>
    <pre>
        #
        #Load training images
        #
        from MnistImageLoader import load_images,ReShapeData

        print("Loading training images")
        (train_data, train_target)=load_images(mnist_train_path_full)
        (train_data1,train_target1)=ReShapeData(train_data,train_target,nb_classes)
        print('Shape:', train_data1.shape)
        print(train_data1.shape[0], ' train images were loaded')

    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="trainingusingpython_createmodel">3-Create the CNN model(TrainMnistFromFolder.py)</h3>
    <pre>
        # 
        # Create a sequential model
        #
        model = Sequential()
        # Add the first convolution layer
        model.add(Convolution2D(
            name="conv1",
            filters = 20,
            kernel_size = (5, 5),
            padding = "same",
            input_shape = (28, 28, 1)))
        # Add a ReLU activation function
        model.add(Activation(
            activation = "relu"))
        # Add a pooling layer
        model.add(MaxPooling2D(
            name="maxpool1",
            pool_size = (2, 2),
            strides =  (2, 2)))
        # Add the second convolution layer
        model.add(Convolution2D(
            name="conv2",
            filters = 50,
            kernel_size = (5, 5),
            padding = "same"))
        # Add a ReLU activation function
        model.add(Activation(
            activation = "relu"))
        # Add a second pooling layer
        model.add(MaxPooling2D(
            name="maxpool2",
            pool_size = (2, 2),
            strides = (2, 2)))
        # Flatten the network
        model.add(Flatten())
        # Add a fully-connected hidden layer
        model.add(Dense(500))
        # Add a ReLU activation function
        model.add(Activation(activation = "relu"))
        # Add a fully-connected output layer - the output layer nodes should match the count of image classes
        model.add(Dense(nb_classes,name="outputlayer")) 
        # Add a softmax activation function
        model.add(Activation("softmax"))
        #
        #Display Summary
        #
        model.summary()

        # Compile the network
        model.compile(
            loss = "categorical_crossentropy", 
            optimizer = SGD(lr = 0.01),
            metrics = ["accuracy"])
        print("Compilation complete");

    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>

    <h3 id="trainingusingpython_trainmodel">4-Train model(TrainMnistFromFolder.py)</h3>
    <pre>
        #
        # Train the model 
        #
        total_epochs=20
        start = time.time()
        model.fit(
            train_data1, 
            train_target1, 
            batch_size = 128, 
            epochs = total_epochs,
	          verbose = 1)
        print("Train complete");
        #
        #Test the model
        #
        print("Testing on test data")
        (loss, accuracy) = model.evaluate(
            test_data1, 
            test_target1,
            batch_size = 128, 
            verbose = 1)

        # Print the model's accuracy
        print("Accuracy="+ str(accuracy))

    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="trainingusingpython_savemodel">5-Save the model (FreezeKerasToTF.py)</h3>
    <p>After training is complete, the model has to be saved in the original TensorFlow format (.pb). 
        The function freeze_session in the file FreezeKerasToTF.py does this for us. The saved model contains the network layout and the weights</p>
    <pre>

        #
        #Saving using Freeze approach https://stackoverflow.com/questions/45466020/how-to-export-keras-h5-to-tensorflow-pb
        #

        frozen_graph = freeze_session(K.get_session(),
                                      output_names=[out.op.name for out in model.outputs])
        tf.train.write_graph(frozen_graph, "Out", "Mnist_model.pb", as_text=False)
        
    </pre>
    <h3 id="trainingusingpython_results">6-Results</h3>
    <img src="images/EpochProgress.PNG"/>
    <br />
    <a href="#toc" title="Back to Table of Contents">Top</a>


    <h2 id="csharpconsole">C# console application</h2>
    <h3 id="csharpconsole_overview">Overview</h3>
    <pre>
        -----------------------
        1)Load trained model file
        -----------------------
                |
                |
        -----------------
        2)Load test images
        -----------------
                |
                |
        -----------------------------------
        3)Evaluate the test image using CNN
        -----------------------------------

    </pre>
    <h3 id="csharpconsole_create">1-Create a Console application</h3>
    <ul>
        <li>
            Create a new Console application using .NET Framework (64 bit, 4.6.1 or above)
        </li>
        <li>
            Add NUGET package reference to TensorflowSharp
        </li>
    </ul>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="csharpconsole_loadmodel">2-Load the trained model file</h3>
    <pre>
        ///
        ///Skeletal code using TensorFlowSharp
        ///
        var modelfile=@"c:\\MyTensorFlowModel.pb";//Produced by training
        byte[] buffer = System.IO.File.ReadAllBytes(modelfile);
        using (var graph = new TensorFlow.TFGraph())
        {
            graph.Import(buffer);
            using (var session = new TensorFlow.TFSession(graph))
            {   
                var file="test.png";
                var runner = session.GetRunner();
                var tensor = Utils.ImageToTensorGrayScale(file);
                runner.AddInput(graph["conv1_input"][0], tensor);
                runner.Fetch(graph["activation_4/Softmax"][0]);

                var output = runner.Run();
                var vecResults = output[0].GetValue();
                float[,] results = (float[,])vecResults;
                ///
                /// Evaluate the results
                ///
                int[] quantized = Utils.Quantized(results);
            }
        }
    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="csharpconsole_imagetensor">3-Utils.ImageToTensorGrayScale</h3>
    <p>This function will load a MNIST picture file and create a TFTensor </p>
    <pre>
        public static TensorFlow.TFTensor ImageToTensorGrayScale(string file)
        {
            using (System.Drawing.Bitmap image = (System.Drawing.Bitmap)System.Drawing.Image.FromFile(file))
            {
                var matrix = new float[1, image.Size.Height, image.Size.Width, 1];
                for (var iy = 0; iy < image.Size.Height; iy++)
                {
                    for (int ix = 0, index = iy * image.Size.Width; ix < image.Size.Width; ix++, index++)
                    {
                        System.Drawing.Color pixel = image.GetPixel(ix, iy);
                        matrix[0, iy, ix, 0] = pixel.B / 255.0f;
                    }
                }
                TensorFlow.TFTensor tensor = matrix;
                return tensor;
            }
        }

    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    <h3 id="csharpconsole_quantized">4-Utis.Quantized</h3>
    <p>This function will convert the TF result into an array with 10 elements. The 0 th element represents the probability for 
        digit 0 and element 9 th represents teh probability for digit 9
    </p>
    <pre>
        //Silly repetitions here! I was running out of time.
        internal static int[] Quantized(float[,] results)
        {
            int[] q = new int[]
            {
                results[0,0]>0.5?1:0,
                results[0,1]>0.5?1:0,
                results[0,2]>0.5?1:0,
                results[0,3]>0.5?1:0,
                results[0,4]>0.5?1:0,
                results[0,5]>0.5?1:0,
                results[0,6]>0.5?1:0,
                results[0,7]>0.5?1:0,
                results[0,8]>0.5?1:0,
                results[0,9]>0.5?1:0,
            };
            return q;
        }
    </pre>
    <a href="#toc" title="Back to Table of Contents">Top</a>
    
    <h3 id="csharpconsole_results">5-Results</h3>
    <p>After iterating over all the 10,000 test images and classifying each of them through MNIST we get a prediction success rate of 98.5%. 150 images were misclassified. 
        As per <a href="http://yann.lecun.com/exdb/mnist/">MNIST home page</a> , the state of the art benchmark is over 99.5% success rate.
    </p>
    <img  src="images/Csharp_Test_Results.PNG"/>
    <br />
    <a href="#toc" title="Back to Table of Contents">Top</a>

    <h2 id="usingthecode">Using the code</h2>
    <h3 id="usingthecode_github">Github Repository</h3>
    <a href="https://github.com/sdg002/MNISTpng">https://github.com/sdg002/MNISTpng</a>
    <h3 id="usingthecode_structure">Solution stucture</h3>
    <pre>
        Solution
        --------
                |
                |
                MNISTPng (ZIP of individual PNG train and test files)
                ------------------------------------------------------
                |
                |
                PythonTrainer (Python script to train using TensorFlow)
                -------------------------------------------------------
                |
                |
                ConsoleAppTester (C# console app using TensorFlowSharp)
                -------------------------------------------------------

    </pre>

    <h3 id="usingthecode_pythontrainer">1-PythonTrainer</h3>
    <p>Python scripts for training a CNN model</p>
    <ul>
        <li><strong>TrainMnistFromFolder.py</strong>-Outermost Python script which loads and trains the images</li>
        <li><strong>MnistImageLoader.py</strong>-Useful for converting a PNG into a Tensor</li>
        <li><strong>FreezeKerasToTF.py</strong>-Useful for saving a trained model as .PB file</li>
    </ul>
    <h3 id="usingthecode_mnistpng">2-MNISTPng</h3>
    <p>ZIP of training and testing images</p>
    <ul>
        <li><strong>testing.zip</strong>-10,000 individual testing files organized into 10 directories</li>
        <li><strong>training.zip</strong>-50,000 individual training files organized into 10 directories</li>
    </ul>
    <h3 id="usingthecode_consoleapptester">3-ConsoleAppTester</h3>
    <p>C# EXE which will use TensorFlowSharp to load the trained model</p>
    <a href="#toc" title="Back to Table of Contents">Top</a>
   

    <h2 id="pointsofinterest">Points of Interest</h2>
    <ul>
        <li><a href="https://www.youtube.com/embed/ArPaAX_PhIs">Andrew Ng tutorials</a></li>
        <li><a href="http://yann.lecun.com/exdb/mnist/">MNIST Home Page</a></li>
    </ul>
    <a href="#toc" title="Back to Table of Contents">Top</a>


    <!-------------------------------    That's it!   --------------------------->
</body>

</html>

